On Teaching Bootstrap Confidence Intervals

notion the theoretical distribution on the random variable ) is not available. All we have ICOTS8 (2010) Invited Paper Engel International Association of Statistical Education (IASE) www.stat.auckland.ac.nz/~iase/ to rely on are the data at hand, i.e. the sample or recapture of size n. These data–if drawn by some random mechanism–may well be taken as a good representation of the total fish population. Implementing the bootstrap idea, we therefore consider samples (=resamples) drawn from the best approximation to the population we may have: the sample. 1. Draw a sample (re-sample or bootstrap sample) of size n with replacement from the original sample and count the number of marked elements in that resample. 2. Compute (bootstrap estimator) 3. Repeat step 1 and 2 many times over to obtain the (empirical) bootstrap distribution of , represented e.g. as a histogram. This distribution – the bootstrap distribution – is the proxy to the unknown sampling distribution of . Example 1: In the simulation experiment described above we “marked” m = 80 fishes and caught a sample of n = 60 animals of which k = 13 had a marker leading to an estimate for the population size of 80 60/13 369. Relying on the available sample of size 60, we resampled repeatedly 500 times to obtain the bootstrap distribution of as displayed in Figure 2 obtaining a 95% percentile confidence interval of [240; 600] by cutting off 2.5% from both tails. (1) For the basic bootstrap interval we obtain [138, 498] while the bootstrap-t interval is [211, 522] Furthermore, it is easy to compute the standard error as the standard deviation of the bootstrap distribution to obtain a value of 78.68. Also, an estimate for the bias of can be computed as difference between the average of the bootstrap distribution and the estimate of in the original sample: 321.058 240 = 81.058 (2) . It is instructive to consider how modern software allows implementing the bootstrap. The histogram in Figure 2 has been created with the educational statistics package Fathom. Figure 2. Bootstrap distribution of the estimated population size: Implementation in Fathom (left) and histogram (right) WHEN DOES BOOTSTRAP WORK: AN ISSUE FOR TEACHING? The bootstrap may be an instructive, very useful and intuitively reasonable algorithm, but does the method really produce reasonable results? Part of the enthusiasm about the bootstrap method was founded on the misunderstanding that mathematics (as the basis in classical inference) is replaced by mere computing power. ICOTS8 (2010) Invited Paper Engel International Association of Statistical Education (IASE) www.stat.auckland.ac.nz/~iase/ Sound statistical inference is based on the sampling distribution of , but with the bootstrap we infer from . To achieve valid conclusions requires that these two distributions are close to each other, at least in some asymptotic sense. Hence, we need a continuity argument to guarantee that the bootstrap is more then a “stab in the dark” (Young 1994). To formalize, we have to show that these two distributions–appropriately normalized– converge to the same limiting distribution. For broad classes of situations this can be proven mathematically, but requires highly advanced methods whose foundation is the theory of empirical processes and their convergence (see, e.g., Hall, 1992; Mammen, 1992). In order to exhibit the required convergence, we resort to the previous example of the capture-recapture estimate. The distribution of marked elements in the resample is a typical case of a hypergeometric distribution while for the Bootstrap distribution of marked elements in the resample we have (drawing with n elements from the resample, where k out of n elements are marked) Now it is straightforward that both distributions, appropriately normalized, converge to the same limit as with . It is instructive to consider elementary examples where the bootstrap method fails. Consider the ”Frankfurt Taxi Problem” (A. Engel, 1987): in Frankfurt taxis are presumably numbered from 1 to N. After arriving at the train station you observe n taxi cabs numbered by . The maximum likelihood estimate for the total number of taxis N is . It is straightforward to show that the distribution of is exponential with parameter which implies in particular that – as for any continuous distribution – the value 0 is assumed with zero probability. The bootstrap distribution for is the distribution of , where for some . We obtain a value of 0 with that probability for which the element xi is being resampled. However, it is well known (compare the “rencontre problem”) that this probability converges towards 1 1 e, hence not towards the value 0 provided by the exponential distribution. 5 SUMMARY It is well known that the concept of confidence intervals is hard for students to grasp. Computer simulation lets students gain experience with and intuition for these concepts. The bootstrap provides a prominent opportunity to enhance that learning in view of genuine statistical reasoning, i.e., in situations where we have data but do not know the underlying distribution. For the mathematical statistician the bootstrap is a highly advanced procedure whose consistency is based on the convergence of empirical processes, for users of statistics it is mainly a simulation method and an algorithm. For a sound understanding and appropriate handling any user should be aware that random and chance enter at two distinct points: in the Plug-In step by considering the distribution of the sampling statistics under in place of the distribution F ICOTS8 (2010) Invited Paper Engel International Association of Statistical Education (IASE) www.stat.auckland.ac.nz/~iase/ resulting in the bootstrap distribution and in the Monte-Carlo step by obtaining an empirical approximation to the exact bootstrap distribution. There are situations that allow to compute the exact bootstrap distribution, i.e., then the Monte Carlo step is not needed. In the vast majority of situations the bootstrap distribution is not tractable analytically. Then simulation from the empirical distribution yields an empirical approximation to the bootstrap distribution. This approximation can be made arbitrarily close by increasing the bootstrap sample size, given the availability of sufficient computing power. In contrast, the asymptotic equivalence of the bootstrap distribution and original distribution of the sampling statistic is far from being trivial and is based on convergence results for empirical processes. A great deal of research efforts on learning and instruction over the last decades focuses on how to take advantage of modern technology to support learning. The availability of modern technology also influences the content of teaching considered valuable and worthwhile. Moore (1997) speaks of synergy effects between technology, content and new pedagogy. Working with technology may influence qualitatively the thinking of learners about mathematics. New content reflects the computer-intensive practice of modern statistics. The bootstrap is a prime example for synergy between technology, content and new insights into the learning process. The method is based on a conceptually simple idea that is generally very useful and instructive. Without available cheap computing power the bootstrap is not feasible. As for any simulations in probability and statistics, the bootstrap can be implemented in an activity-based, exploratory and experimental learning environment. While the method in its first three decades has been mainly a very useful method for the expert data analyst, time has come to take advantage of its great potential to enhance learning of concepts in inferential statistics. NOTES(1) Based on the hypergeometric distribution for the number k of marked elements, it is herepossible to compute the exact 95% confidence intervals of [218; 480] while a probabilitysimulation (based on the total population) resulted in a 95% confidence interval of [228; 485].(2) The true population size in the simulation example is N=310. REFERENCESBatanero, C., Biehler, R., Engel, J., Maxara, C., & Vogel, M. (2005). Using Simulation to BridgeTeachers Content and Pedagogical Knowledge in Probability. In: ICMI-Study 1, Online:http://stwww.weizmann.ac.il/G-math/ICMI/log_in.html.Davison, A. C., & Hinkley, D. V. (1997). Bootstrap Methods and their Applications. CambridgeUniversity Press.DiCiccio, T. J., & Efron, B. (1996). Bootstrap Confidence Intervals (with discussion). StatisticalScience, 11(3), 189-228.Efron, B., & Tibshirani, R. J. (1993). An Introduction to the Bootstrap. London: Chapman & HallEfron, B. (1982). Bootstrap methods: another look at the jackknife. Annals of Statistics, 7, 1-26.Engel, J. (2000). Markieren Einfangen Schätzen: Wie viele wilde Tiere? Stochastik in derSchule, 2, 17 24.Engel, J. (2007). On Teaching the Bootstrap. Bulletin of the International Statistical Institute 56th Session, Lisbon.Hall, P. (1992). The Bootstrap and Edgeworth Expansion. New York: Springer.Hesterberg, T. (1998). Simulation and Bootstrapping for Teaching Statistics. In AmericanStatistical Association: Proceedings of the Section on Statistical Education, 44 52.Mammen, E. (1992). When does bootstrap work? Asymptotic results and simulations Heidelberg:Springer.Moore, D. (1997). New pedagogy and new content: The case of statistics. International StatisticalReview, , 123-166.Young, A. (1994). More than a stab in the dark? Statistical Science, 9, 382-415.